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Key messages 



A A systematic review of research is a decision-making tool for policy and practice. It is a piece of 
research in its own right, using explicit and rigorous methods that follow a standard set of stages. 
These methods identify, critically appraise and synthesise relevant research (both published and 
unpublished) around a specific research question. 

A The review process allows for different studies to be weighted for relevance and quality of 
findings to answer a given question. The ultimate effect of this is that research can influence a 
review’s conclusion only when based on agreed guidelines, and when the reviewers have 
confidence in the research. 

A In undertaking the first systematic review of research in vocational education and training 

(VET) in Australia on the mature-aged and skill development activities, the National Centre for 
Vocational Education Research (NCVER) was required to also establish a model and 
infrastructure for future reviews. NCVER’s proposed eight-step model is outlined in this report. 
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Executive summary 



The National Centre for Vocational Education Research (NCVER) was contracted by the 
Australian National Training Authority (ANTA) to undertake a first systematic review of research 
related to the topic of mature-aged workers. The contract included the development of a replicable 
framework and infrastructure for further systematic reviews of research. 

A systematic review of research is a decision-making tool for policy and practice. It is a piece of 
research in its own right, using explicit and rigorous methods that follow a standard set of stages. 
These methods identify, critically appraise and synthesise relevant research (both published and 
unpublished) relating to a specific research question. 

In undertaking the first systematic review of research in vocational education and training (VET) 
in Australia, NCVER learnt many lessons. Our eight-step model and developed infrastructure for 
future reviews includes: 



Step 1: Identify the review question 

A A steering group will be established and include high-level representation from state, territory 
and Commonwealth authorities, from industry and from the research community, or specific 
experts for the particular topic. 

A A consultation group will be established and include external reviewers and potential reviewers 
or those with expertise, or who have expressed interest in the topic (including international 
advisors). 

A Policy-makers and other stakeholders will be involved in defining the review question by 
focusing on a very specific population, intervention and outcome. 

A Sufficient time will be allowed to consult widely with key groups and individuals for refinement 
of the question. 

A The key reviewer and second reviewer will be selected at the beginning of the review process. In 
this way they will fully understand the development of the question, be involved in screening 
the studies for in-depth review, and be familiar with all included studies before synthesising the 
evidence and compiling the final report. 



Step 2: Develop a framework document 

The framework document takes the review question and defines keywords, search strategy, review 
and appraisal criteria, and contents of final report. 

A The advice of the consultation group will be used to guide the development of the framework 
that arises from and supports the question. 

A The key reviewer will contribute to the management of the review process (for example, 
communicating with consultant reviewers). 
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A Once the framework is established, a database will be developed to contain the results of 

searches, critical appraisal and selection (using inclusion and exclusion criteria) of materials, and 
the relevant findings and evidence from the included studies to answer the review question. 



Step 3: Search for all relevant research 

A At least two searchers will undertake the extensive and thorough searching process for each review. 

A The initial selection will be done by the searchers, using titles and abstracts, and should include 
all material that appears to meet the inclusion criteria established within the framework, and if 
in doubt, they should include the material. 

A The steering and consultation group will be provided with lists of the excluded and included 
studies and consulted to ensure that no key studies are missed. However, the final decision on 
inclusion will remain with the project team. 

A Screening is an iterative process and a further screening stage (see next step) using full 

documents will be undertaken by the key reviewer and second reviewer to arrive at the final 
selection of studies for in-depth review. 



Step 4: Select studies to be included 

A The inclusion criteria will be applied more strictly to the full documents by the key reviewer and 
second reviewer who will be involved in the final screening stage. This will include an initial 
appraisal of the relevance of findings to the review question and of the quality of the research. 

A Only research studies which provide evidence to answer the review question and which meet the 
quality criteria will be included in the in-depth review. There should be no more than 20 studies 
included for in-depth review (there may be a set of ‘reserve’ documents or lower rated studies 
kept for contextual information). 

A The reviewers who will be synthesising the evidence for the report will be familiar with all the 
included studies and aware of those excluded from the in-depth evaluation. 

A Sufficient time will be allowed so that references within the selected studies can be followed up, 
in order that these may be considered for inclusion in the review process. 

A All selected reviewers will attend a training workshop before commencing the in-depth appraisal 
and review of evidence from the selected studies. Detailed guidelines will be provided to the 
reviewers. 



Step 5: Appraise the studies 

A Reviewers should be allocated studies, to some extent, according to their expertise in both the 
topic and the research and analytical techniques, as required (for example, quantitative research, 
including statistical analysis and economic modelling or qualitative research). 

A Each study included in the in-depth review will be allocated to two reviewers who, working 
independently, will enter into the electronic template their appraisal of the relevance and quality 
of the findings to the review question. The reviewers will then reach a consensus decision. 

A The project team may moderate ratings given by reviewers where consensus is not possible or 
some inconsistency is noted. 

A The reviewers will add to the database the details of study aims, methods, population, 
intervention, outcomes and findings and the best examples to illustrate the findings. 
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Step 6: Synthesise the evidence 

A A database will be used in future to enable electronic sorting and amalgamation of evidence 
from the studies to assist in the synthesis of findings and in checking the evidence trail for the 
final report. 

A The key reviewer and a second reviewer will synthesise the evidence found to answer the review 
question into categories, pooling material from the studies in whose findings we can have 
confidence. 

A Using other members of the project team to provide feedback, the key reviewer will compile a 
draft final report of the evidence to answer the review question, and implications for policy, 
practice and research. 



Step 7: Present findings to stakeholders 

A The draft final report will be distributed to the steering group members and reviewers for 
comments before finalising for publication. 



Step 8: Disseminate the findings 

A The final report will be published on the NCVER website. 

A Presentation of the findings of systematic reviews will be made to stakeholders through research 
forums, conferences and other channels as appropriate. 
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Introduction 



A systematic review can be defined as a review of a clearly formulated question that attempts 
to minimize bias using systematic and explicit methods to identify, select, critically appraise 
and summarize relevant research. (Needleman 2002, p.6) 



What is a systematic review of research? 

A systematic review is a decision-making tool for policy and practice. It uses explicit and rigorous 
methods to identify, critically appraise and synthesise relevant research (both published and 
unpublished) around a specific research question. 

A systematic review is a piece of research in its own right, using explicit and transparent methods 
that follow a standard set of stages. This enables it to be replicated. It is also undertaken by a team 
and the outcome is a collective one — which reduces potential bias. 

The review process allows for different studies to be weighted for relevance and quality of findings to 
answer a given question. The ultimate effect of this is that research can influence a review’s conclusion 
only when based on agreed guidelines, and when the reviewers have confidence in the research. 

A meta-synthesis uses textual analysis to synthesise findings from qualitative research studies and 
those quantitative research studies where numerical data cannot be combined. A meta- analysis, on 
the other hand, is a ‘the statistical analysis of a large collection of results from individual studies for 
the purpose of integrating the findings’ (Glass 1976). The synthesised findings are usually presented 
in the form of a structured narrative, with tables summarising the findings from the reviewed studies. 

Based on the work of the Evidence for Policy and Practice Information and Co-ordinating Centre 
(EPPI-Centre‘) in the United Kingdom, the National Education Research Forum^ summarises 
systematic reviews in its ‘Advice and information for funders’. They note that: 

^ They are pieces of research in their own right using explicit and transparent methods and 
follow a stand set of stages. This enables them to be replicated. 

^ The process allows for different studies to be weighted for quality and relevance of 
evidence for a given question. 

^ The process produces a map of evidence which helps classify the research. 

^ They are undertaken by teams and the outcome is a collective one — which reduces 
potential bias. 

^ The process allows for reviews to be updated — even by different authors — and so provides 
flexibility and value for money in the longer term. 



‘ The EPPI-Centre is part of the Social Science Research Unit, Institute of Education, University of London. Since 
February 2000 the EPPI-Centre has been funded by the Department for Education and Skills (DIES) to provide a 
resource for those wishing to undertake systematic reviews of educational research, and for those wishing to use reviews 
to inform policy and practice. Detailed information available from <http://eppi.ioe.ac.uk/>. 

^ A website funded by the Department of Education and Skills, London. Information available from <http://www.nerf- 
uk.org/>. 



12 



An aid to systematic reviews of research in vocational education and training in Australia 





^ The review process is designed to support user engagement e.g. practitioners taking part in 
undertaking reviews. 

^ Some systematic review methods enable qualitative and quantitative studies to be analysed 
and compared in the same review. 

^ Participating in a systematic review helps improve research skills and can help researchers 
address how they report on primary research. 

^ The systematic review process is criterion-based, transparent and public. 

^ Systematic reviewing enables international collaboration and supports inclusion of 
international evidence in a review. 

(National Education Research Forum website, p.l) 



Background 

In the 1970s, systematic reviews of research were pioneered in health care by the Cochrane 
Collaboration, which linked research and development sites across the world to review and analyse 
randomised clinical trials from an international perspective. From these reviews Cochrane generated 
reports to inform practitioners, to influence practice and to be a resource in the development of 
consensus guidelines. Essentially, ‘evidence-based practice as it relates to health care is the 
combination of evidence derived from individual clinical or professional expertise with the best 
available external evidence to produce practice that is most likely to produce a positive outcome for 
a patient or client’ (Pearson 2004). However, as Sackett (1989) explains, ‘the nonexperimental 
evidence that forms the recalled experiences of practitioners with expertise will tend to overestimate 
efficacy’. Sackett outlines three reasons for this: favourable treatment responses are more likely to be 
remembered; unusual patterns of symptoms when reassessed, even a short time later, tend to return 
toward a more usual, normal result; and, both patients and their clinicians have a desire for 
treatment to be successful which can cause both parties to overestimate effectiveness. 

Knowledge acquired from qualitative approaches to research has been largely absent from the 
Cochrane Collaboration systematic reviews (Pearson 2004). As Pearson notes, the ‘development of 
accepted approaches to the appraisal and synthesis of evidence’ by those with expertise in qualitative 
approaches to inquiry has been slower than that by quantitative researchers. 

Qualitative research is centrally concerned with understanding things rather than measuring them. 
Qualitative research is best used for problems where the results will increase understanding, expand 
knowledge, clarify the real issues, generate hypotheses, identify a range of behaviours, explore and 
explain motivations, attitudes and behaviours, identify distinct behavioural groups, or provide input 
to a future stage of research or development (Gordon & Langmaid 1988). 

During the 1990s, the development of approaches to include qualitative evidence in systematic 
reviews of research has become important in health care (Pearson 2004), social science (Spencer 
et al. 2003), and education (Evidence for Policy and Practice Information and Co-ordinating 
Centre 1993). 

Systematic reviews and vocational education and training (VET) 

While the scientific or experimental approach is considered to ensure objectivity or non-bias in the 
findings of medical research for systematic reviews, we note that, in social science research, the use 
of control and treatment groups for comparative purposes is rare and often impractical or unethical. 
A single intervention is going to be difficult to isolate, as usually a multitude of factors impact on an 
individual’s choice to undertake vocational education and training (VET) or on the outcomes of 
VET. Thus, determining a direct link between an intervention and outcome is more difficult in 
vocational education and training than in heath care with its randomised controlled trials. 

Dawe (2003) notes that a broad definition of research is used for VET research in Australia: a 
‘systematic and organised way of finding answers to questions’. In addition, VET research often 
comprises a combination of different research methods. 
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In establishing systematic reviews of research and reviewer groups, the National Centre for 
Vocational Education Research (NCVER) is intending to encourage an informed and robust 
critique of research related to policies and practice in VET and to bring research directly into the 
decision-making processes at the level of both policy and practice. 

In the associated report (Thomson et al. 2005), a narrative summary is used in reporting the 
findings of the first systematic review that NCVER has undertaken. As noted by Greenhaigh et al. 
(2004) in analysing different research approaches, a narrative, rather than statistical, summary 
technique is used. 

Because different researchers in different traditions had generally conceptualised the topic 
differently, asked different questions, privileged different methods, and used different criteria 
to judge ‘quality’ and ‘success’, we used narrative, rather than statistical, summary techniques. 

We highlighted the similarities and differences between the findings from both an 
epistemological and an empirical perspective. In this way, heterogeneity of approaches and 
contradictions in findings may be turned into data and analysed systematically, allowing 
conclusions to be drawn that go beyond statements such as, ‘the findings of primary studies 
were contradictory’ or that ‘more research is needed’. (Greenhaigh et al. 2004, p.8) 



Why do a systematic revie’w? 

The aim of a systematic review is to develop a concise summary of the best available evidence that 
addresses a clearly defined question in a particular area of interest. While the question addressed by 
a systematic review may be the same as that posed by a primary researcher, the difference is that 
primary research must exist on the topic to make conducting a review worthwhile (Averis & 
Pearson 2003). For further details on the differences between systematic reviews and traditional 
literature reviews, and common misconceptions of systematic reviews, we suggest reading 
Needleham (2002) and Petticrew (2001). 

‘Without a requirement that reviewers clearly specify inclusion criteria and then exhaustively 
include all studies that fit these criteria, reviewers may consciously or unconsciously decide to 
include studies that favour their own biases and ignore those that do not’ (Slavin 1995). This does 
not mean that value judgements can be totally isolated from the systematic review process, but 
rather that reviewers have to be explicit about procedures in creating the review, giving readers 
enough information about the studies so they can gain a good understanding of the original 
research (Slavin 1995). Reviewers must assess the quality of the study by examining how far its 
design, content and analysis have helped or hindered the minimisation of bias. Reviewers must be 
at pains to carefully and critically appraise the evidence, because the inclusion of biased studies can 
provide misleading final results. It should be noted that the quality of studies is on a continuum 
and includes the two main criteria of research design and implementation; the design needs to fit 
the purpose of the study and be well conducted. 



What ’we did 

NCVER was contracted by the Australian National Training Authority (ANTA) to undertake a 
systematic review of research related to the topic of mature-aged workers. To the best of our 
knowledge this was the first systematic review conducted in Australia in the field of vocational 
education and training. As part of the contract, NCVER was to develop a replicable framework and 
infrastructure within which further systematic reviews of research could be conducted. 
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The starting point for the NCVER researchers was to familiarise themselves with models already 
being used for systematic reviews of social and educational research overseas. Four key organisations 
provided models for investigation; 

A the Cochrane Collaboration which pioneered systematic reviews in the field of medicine, 
establishing what interventions work and for whom 

A the Campbell Collaboration in the United Kingdom which developed a model for the social 
sciences to mirror that of the Cochrane Collaboration 

A the Evidence for Policy and Practice Information and Co-ordinating Centre in the United 
Kingdom which supports external review groups by providing tools and procedures to assist 
groups to undertake and disseminate systematic reviews of research of social interventions in 
health and education 

A the Learning and Skills Development Agency in the United Kingdom which is a strategic 
national resource for the development of policy and practice in post- 16 education and training. 
The Learning and Skills Development Agency uses the Evidence for Policy and Practice 
Information and Co-ordinating Centre model and has also developed its own hybrid model. 

Other relevant research reports were also used in developing this new NCVER systematic review 
process; for example, Spencer et al.’s Quality in qualitative evaluation: A framework for assessing 
research evidence (2003); Averis and Pearson’s Filling the gaps: Identifying nursing research priorities 
through the analysis of completed systematic reviews-, Pearson’s Balancing the evidence: Incorporating the 
synthesis of qualitative data into systematic reviews (2004); Glass, McCawand Smith’s Meta-analysis 
in social research (1981); Slavin’s Best evidence synthesis: An intelligent alternative to meta-analysis 
(1995); and Sackett’s Rules of evidence and clinical recommendations on the use of antithrombotic 
agents (1989; Cook, Guyatt, Laupacis & Sackett 1992). 

The NCVER process for a systematic review was further developed with feedback from the steering 
group, consultation group, and ten external reviewers selected from expressions of interest for this 
first systematic review. 



Outline of this report 

This report reflects the process undertaken in 2004 in developing our own model for systematic 
reviews of research in vocational education and training in Australia. The associated report 
(Thomson et al. 2005) outlines what we found in relation to the first review on the topic of 
mature- aged workers. The first systematic review will be updated in 2005 using the refinements to 
the review process as outlined in this report. 

The following chapters discuss the process undertaken for each step in the NCVER model and 
what was learnt from this process to aid future systematic reviews. The eight steps identified for the 
NCVER model are: identifying the question; developing a framework; searching for studies; 
selection of relevant studies; appraising quality of selected studies; synthesising the findings; 
presenting and disseminating the findings of the systematic review to stakeholders. 
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Identifying the question 



The first step in a systematic review is to define a focused question. The focused question 
provides the same role and is as important as the research question or hypothesis in primary 
research. The focused question addresses an important and relevant question. Narrowing the 
focus produces an answerable research question. Otherwise, the question might be too broad 
to have any chance of being answered or could in fact be a series of questions. Whilst the 
narrowing of scope in a systematic review may be perceived as a disadvantage compared with 
traditional reviews ... it helps to ensure the review will provide as conclusive a summary as 
data permits. (Needleman 2002, p.6) 

What we did 

Involvement of stakeholders 

Defining the review question is the most critical part of the review because other aspects of the 
process flow directly from this. The question needs to be relevant to policy-makers and to have a 
body of research behind it. It should also be one that does not require an immediate answer, as a 
review will typically take up to a year. 

For the first systematic review, NCVER convened a meeting of a steering group which included 
high-level representatives from state and territory authorities, the Australian Government, ANTA 
and the research community. At this meeting, the group considered papers developed by the 
NCVER project team providing a general introduction to systematic reviews and examples of 
systematic reviews which were sourced from the five organisations listed in the previous chapter 
under ‘What we did’. The group was also provided with an overview of the systematic review 
process, and taken through an actual example. 

Generally, the recognised format to focus a systematic review question is to consider the 
population of interest, intervention for investigation and outcomes considered most important for 
assessing the results. 

The group discussed the definition of and current policy issues around the topic of ‘mature-aged 
workers’, and relevant research being undertaken in this area. The question was then refined from 
these discussions. 

Population 

Consideration was given to looking at all ages since initial human capital formation affects labour 
market attachment over time. Looking at all those over 25 years was also considered. In the final 
instance, it was agreed that this review should focus on those people 45 years of age and over 
because they are of immediate policy relevance. On current trends in demography and workforce 
participation, there will be a significant shortage of labour by the end of the decade of the order of 
1.4 million workers, if those people over 45 years do not increase and extend their labour market 
participation rates. 
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Interventions 

At the time of forming the question, the Organisation for Economic Co-operation and 
Development (OECD) was looking at all factors impacting on extending the working life of people 
50 years of age and over in Australia (and 19 other OECD countries). Considering that the VET 
sector’s core business is education and training, it was decided to restrict the interventions of 
interest to learning programs that were eventually referred to as skill development activities. 

Outcomes 

Since the focus of VET is to enhance employability, it was decided to focus on labour market 
outcomes rather than broader social and economic outcomes, which might also include working in 
the voluntary sector or improving health and wellbeing of older people. However, it was agreed that 
the outcomes should go beyond just extending the working lives of existing workers (scope of 
OECD study) to include increasing the total numbers of people over 45 years attached to the 
labour market. In addition, it was agreed that the outcomes would include increased productivity 
while at work. 

Using an iterative process, the draft final question was defined, noting the need to limit the scope to 
make it tractable, and that policy-makers are particularly interested in interventions (what works 
and what does not work). 

The draft question was defined as: 

What evidence is there that skill development activities for mature-age workers (45 years 
and over) lead to: 

improved attachment to the labour market? 
dr improved productivity? 

Implied in the review question, and to be reported in the research findings and final report, were 
the issues of: 

dr the factors that have an important bearing on these key outcomes, either as barriers or facilitators 
dr implications for policy, practice and research of the findings. 



Defining the terms in the question 

After drafting the question, the terms in the question were defined by the steering group: 

Defining mature-aged 

The starting brief contained the term ‘mature-aged workers’, but after discussion by the steering 
group, the population was defined more broadly as ‘mature- aged’. 

Mature-aged was defined as all people 45 years and over. This was to include all those who desire 
work, or who might desire work if their skills were more suitable or if there were more opportunities. 

Defining skill development activities 

Skill development activities were defined as ‘deliberate’ general education or specific vocational 
activities undertaken to learn new skills or further develop skills related to work. ‘Deliberate’ in this 
context referred to the person knowingly participating in the skill development activity, for 
example, an activity which has a predetermined plan and format designed to develop employment- 
related skills and competencies. 

Skill development activities thus included a structured (formal) learning activity, such as 
undertaking an accredited or non-accredited course, or as an informal learning activity, such as self- 
directed learning, networking, coaching or mentoring, as long as it was predetermined or 
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knowingly participated in. Skill development activities could have also been initiated by self, 
employer or other (such as, Job Network member). 

Improved attachment to the labour market 

Improved attachment to the labour market included: 

A increased labour force participation rate 
A increased employment duration 

A job retention or improved job security (no longer at risk of losing job) 

A change of work role or new career following new interest (including self-employment) 

A change in attitude to work, ‘early retirement’ or ‘retrenchment’, for example, looking forward to 
continuing participation in paid work past 55 years. 

Defining improved productivity 

Improved productivity included: 

A increased wages 
A promotion 

A increased efficiency in processes and work output 

A reduction in accidents and injuries to workers through improved health and safety. 

The steering group noted that outcomes did not include participation in the voluntary sector. 



Defining the scope of the research to be included 

The search strategy set by the steering group included all studies (primary and secondary research, 
policy documents and articles) that: 

A were in English (from Australia and overseas) 

A were recent (from approximately the last ten years) 

A matched keywords deemed relevant to the review question. 

The scope also included research that dealt with a population under 45 years where the research 
findings were relevant to improving attachment to the labour force for the mature-aged. Similarly, 
while research was likely to cover those up to 65 years, relevant research focusing on over 65-year- 
olds was also included. 

Feedback on the draft question 

The draft question and definitions of terms determined by the steering group were then sent to a 
consultation group within the VET community for comments and suggestions for improvement. 
Feedback received included the need to note which skill development activities worked, under what 
circumstances, for which group of mature-aged, and what needed to change in current policy and 
practice. It was also considered important to record what other factors (such as employer and 
employee attitudes) influence the key outcomes of improved labour market attachment and 
productivity levels. 



What ’we learnt 

Focusing on the population, intervention and outcomes worked well as the starting point for defining 
the review question in this systematic review. Defining the review question was an iterative process 
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involving policy-makers until the scope of the question was sufficiently narrow to be useful, but 
sufficiently broad to ensure that there were relevant research studies on the topic. 

Our key learning from this phase was that it is crucial to allow sufficient consultation time with key 
groups and individuals for the development of the question. 

In NCVER’s ongoing program of research, a steering group will be established to oversee all 
systematic reviews. The group will have high-level representation from state, territory and 
Commonwealth authorities, from industry and from the research community. It will invite 
membership from individuals with expertise in the particular topic area chosen for each review. 

For each review a consultation group will also be established. Members of the consultation group 
will also have expertise in the relevant topic area and in the research and analytical techniques, as 
required (for example, quantitative research, including statistical analysis and economic modelling 
or qualitative research). Others will include those who have expressed interest, such as those with 
experience of systematic reviews in other disciplines or those undertaken by the Learning and Skills 
Development Agency, in addition to a pool of experienced or potential reviewers. 



Conclusions 

Step 1 : Identify the review question 

A A steering group will be established and include high-level representation from state, territory 
and Commonwealth authorities, from industry and from the research community, or specific 
experts for the particular topic. 

A A consultation group will be established and include external reviewers and potential reviewers 
or those with expertise, or who have expressed interest in the topic (including international 
advisors). 

A Policy-makers and other stakeholders will be involved in defining the review question by 
focusing on a very specific population, intervention and outcome. 

A Sufficient time will be allowed to consult widely with key groups and individuals for refinement 
of the question. 

A The key reviewer and second reviewer will be selected at the beginning of the review process. In 
this way they will fully understand the development of the question, be involved in screening 
the studies for in-depth review, and be familiar with all included studies before synthesising the 
evidence and compiling the final report. 
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Developing a framework 



What we did 

The framework 

The second step of the systematic review of research is the development of the framework for the 
review. The framework for the review presents the background to the project, the scope of the 
review and an outline of the review process to the VET community and general public. 

NCVER’s first framework included the defined policy question, the definitions of the terms in the 
question, the criteria to be used to select research studies for inclusion in the review, the coding to 
be used to record the findings from the included studies, and the criteria for appraising the quality 
of the included studies. It also detailed the content for the final report. A list of the steering group 
members, NCVER project personnel, and external reviewers, and an outline of the review process 
were also provided in the framework document for reference (see appendix A). For further 
feedback, the draft framework was sent to the steering and consultation groups with the finalised 
policy question, and at this point the framework was finalised. 

The evaluation template 

From the finalised framework, a draft evaluation template was developed by the NCVER project 
team. This template was designed for reviewers to include, in detail, the aims, research methods and 
findings from the research studies to provide the information for synthesis in the final report (see 
appendix D). It was designed with specific codes for the relevant components of the evidence; for 
example, details of the population involved and types of interventions and outcomes. It was 
envisaged that the coding would be a useful tool in synthesising the findings by searching in a 
proposed database of the reviewed studies. 

The reviewers’ evaluation template captured the evidence for the review question and also provided 
the criteria by which each study was appraised for relevance and quality of findings to answer the 
review question. 

NCVER trialled the draft template in an internal pilot using three different studies and six different 
internal researchers. It was amended and presented at the training workshop for trial and discussion 
with the external reviewers. Following feedback at this workshop, a number of changes were made 
and the final template was sent electronically to reviewers to use on their first study. During this 
process further clarification was sought by reviewers and NCVER’s responses were circulated to all 
reviewers before they proceeded to their next study. This feedback process continued to support 
reviewers throughout the review process. 



What we learnt 

It is useful to have the advice of the consultation group as well as the steering group to guide the 
development of the framework that arises from, and supports the review question. 

The person(s) appointed to undertake the final synthesis of the evidence should be involved in the 
review process from this point onwards to enable them to fully understand the background to the 
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review question. This key reviewer needs also be involved in the final selection of studies to enable a 
better understanding of the scope of all the studies selected, as well as those studies included for in- 
depth review by the reviewers. 

For each review, careful consideration is required in relation to documents to be included for in- 
depth review. Policy documents were not useful in the first review because they did not provide 
evidence to answer the review question and generally were based on an assumption that training 
would lead to improved attachment to the labour market. Such narrative opinion and text may be 
useful in the screening process to confirm references which may contain evidence to answer the 
review question. Fiowever, when available, only studies which provide evidence or new analysis of 
literature or data should be included for the in-depth reviews. 

Additional time should be allowed to check new primary references which are provided in selected 
studies in order to include these in the review process also, where appropriate. 

The use of a database as a receptacle for the completed in-depth reviews is recommended for future 
reviews. While it was the intention that the completed reviewers’ evaluations be imported into a 
database, time did not permit the development of the database during the first review. The database 
will assist in the transparency of the review process by providing a clear record in one location of all 
the review activity. During the synthesis phase, the database will also assist reviewers in managing 
the categorisation and grouping of findings into synthesised findings. 



Conclusions 

Step 2: Develop a framework document 

The framework document takes the review question and defines keywords, search strategy, review 

and appraisal criteria, and contents of final report. 

A The advice of the consultation group will be used to guide the development of the framework 
that arises from and supports the question. 

A The key reviewer will contribute to the management of the review process (for example, 
communicating with consultant reviewers). 

A Once the framework is established, a database will be developed to contain the results of 

searches, critical appraisal and selection (using inclusion and exclusion criteria) of materials, and 
the relevant findings and evidence from the included studies to answer the review question. 
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Searching for all relevant research 



A systematic review is a piece of research. Like any piece of research, it uses research methods 
that aim to make it produce valid and reliable results. For example, systematic reviews include 
efforts to find as much as possible of the research which addresses the review’s research 
question. This is important if the review’s conclusions are not to be over-influenced by studies 
which are simply the easiest to find. 

(Evidence for Policy and Practice Information and Co-ordinating Centre website) 

What ’we did 

The search strategy 

The third step of the systematic review is the searching phase. A coherent search strategy was 
required to identify all potentially relevant studies to the review. The search strategy was recorded 
in detail so that the process was transparent and replicable. 

The systematic searching for relevant literature was undertaken during May and June 2004. It used 
a range of identified resources, including electronic online databases, web-based databases, web 
search engines, websites and hand-searching of bibliographies or reference lists and journals — hard 
copy and content pages online (see appendix B). 

In developing the search strategy for electronic and web-based databases, not only did we identify 
key terms, but also a range of synonyms (see appendix C). For example, we included variations in 
terminology and in spelling, such as ‘labour’ and ‘labor’, ‘ageing’ and ‘aging’. We also used singular 
and plural of terms such as ‘older worker’ and ‘older workers’. 

During the searching phase, the steering group and experts in the field of mature-aged research 
were consulted to identify other sources of published and unpublished relevant research. The choice 
of databases and key search terms was submitted to the steering group as part of the draft 
framework prior to implementation. The selection process was piloted on a subset of primary 
studies. A training session was run with ail those selecting the studies to ensure uniformity of 
selection. 

Studies initially included in this review were selected on the basis of the criteria listed in table 1 . 

The search log 

A log was kept of the databases and websites searched to assist the reporting of the search process. 
The selection of research studies was largely based on abstracts and titles. The number of relevant 
studies found, the key search terms and the number of records matched was documented. The 
search log template is shown in table 2. 
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Table 1 : Criteria used to select studies for the systematic review 



Item 


Criteria 


Language & geographical limits 


English, from Australia and overseas 


Date of research study 


Approximately the last 10 years 


Scope 


For a study to be included it had to meet the research study, population and 
intervention criteria, and at least one of the outcome criteria. Where 
population was ‘mature-aged’, intervention was ‘skill development activities’, 
and outcome was either ‘improved attachment to the labour market’ or 
‘improved productivity’ 


Key websites 


As listed in appendix B 


Key databases 


As listed in appendix B 


Key authors 


Recommendations were sought from steering group and topic experts 


Key papers/journals 


Recommendations were sought from steering group and topic experts 



Table 2: Search log template 



ID range 


Key 


No. 


No. 


No. 


Down 


Database 


Date 


Searcher 


of records 


search 


records 


records 


matched 


loaded file 


or website 






added to 


terms 


displayed 


selected as 


records 


saved as 


name 






SYSREV 


used 




possible 

inclusions 













Each individual study was assigned a specific ID number as it was recorded in the search log and 
these ID numbers stayed with the particular studies throughout the process and in the final report. 

The searching approach applied by NCVER for this systematic review is summarised in figure I. 

Figure 1 : Summary of the searching approach 




Note: * SYSREV is the name given to the database NCVER deveioped for the systematic review using DB/TextWorks®. 
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Search results 

Search results were screened through an iterative process. In the first instance, the total of 201 1 
references from the initial searches on keywords was reduced through selection based on title or 
abstract of those that appeared to match the basic selection criteria; that is, were in English and 
had been produced between 1994 and 2004. These 548 references were then further reduced to 
those 85 studies deemed to match the inclusion criteria for the review question — population, 
intervention and outcome(s). The search results for these two stages are summarised in figure 2. 

The set of 85 references from the literature searches deemed to match the inclusion criteria were 
entered into the SYSREV database in readiness for the selection process to determine the final set 
of references to use for the review. Reasons for inclusion or exclusion were recorded in the 
SYSREV database. 

Following the initial search process, there was a second screening or selection phase where studies 
were screened based on a more stringent application of the inclusion criteria relating to the study 
itself (as described in the next step — Selecting studies to be included). 



Figure 2: Searching process — summary of stages one and two 

Keyword searching yields 
2011 studies 



Excluded on title or 
abstract because did 
not have intervention 
or outcome 




Excluded 1463 



548 



Interrogation of abstract 
or paper for appropriate 
population, intervention 
and outcome 




Excluded 463 



85 



What we learnt 

At least two searchers are needed to undertake the extensive and thorough searching process for 
each review. Searchers need to include all material that appears to meet the inclusion criteria 
established within the framework, and if in doubt they should include it. 

A further screening process (see next step) must be applied to arrive at the final selection of 
appropriate studies for in-depth review. 

Sufficient time must be allowed to follow up references within the selected studies. 

It is useful to store the information about both included and excluded studies in a structured 
database so that reports can be easily generated and attributes (such as reasons for exclusion and 
findings and evidence) can be easily recorded and retrieved. 



Conclusions 

Step 3: Search for all relevant research 

A At least two searchers will undertake the extensive and thorough searching process for each review. 
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A The initial selection will be done by the searchers, using titles and abstracts, and should include 
all material that appears to meet the inclusion criteria established within the framework, and if 
in doubt, they should include the material. 

A The steering and consultation group will be provided with lists of the excluded and included 
studies and consulted to ensure that no key studies are missed. However, the final decision on 
inclusion will remain with the project team. 

A Screening is an iterative process and a further screening stage (see next step) using full 

documents will be undertaken by the key reviewer and second reviewer to arrive at the final 
selection of studies for in-depth review. 
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Selecting studies to be included 



What we did 



The fourth step in the systematic review is the selection of studies for inclusion in the reviewing 
process. This was an iterative process undertaken by the project team in consultation with the 
steering and consultation groups, including the reviewers. 

Initially, the project team assessed the 85 studies or documents for relevance against the definitions 
of population, intervention and outcomes in the review question as outlined in table 3. At the 
request of the steering group, both primary and secondary research studies were included for this 
review. An initial set. List A, consisting of 37 studies considered to meet the criteria (see table 3) 
was circulated to the steering and consultation groups along with the remainder in List B consisting 
of 48 potentially excluded studies. 

Useful feedback was received and some adjustments were made to the initial sets (see figure 3), after 
which List A consisted of 32 studies to be included in in-depth evaluation by reviewers. A number 
of studies initially excluded on the basis that they did not address the precise review question were 
included. This was because they appeared to have substantial information about barriers or 
facilitators to skill development activities for the mature-aged, which the steering group had 
requested be considered within the scope of the review. 

As reviewers examined the full text of the documents in detail, other references were discovered 
which required follow-up. One study was added to the selected studies in this way. Another study 
was included because a pre-publication final draft became available soon after the allocation of 
studies to reviewers. Thus the final A list consisted of 34 studies. For one of these the full document 
could not be obtained. The remaining 33 studies were evaluated by the reviewers. 

Figure 3: Summary of selection process for studies to be included for in-depth review 



85 studies 



V s 

\ s 




37 List A 



85 studies split into ‘A’ (most 
preferred) and ‘B’ list 



48 List B 



V 

32 List A 



30 allocated to reviewers. 
Others reviewed in internal 
pilot or workshop, or 
allocated subsequently 



V 

34 List A 



Excluded 1 1 and added 6 through 
scrutiny of abstract and suggestions 
by a wider audience. Population, 
intervention and outcome recorded 
for each 



1 added, newly published. 

1 added from bibliography of 
reviewed studies 
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Table 3: Inclusion criteria used to select studies for the reviewing process 



Key term 


Description 


Criteria 


Population 


Mature-aged 


Mature-aged is defined as persons aged 45 years or over. 

Research should also be included where the findings are relevant to those aged 45 
years or over. 


Population 


Worker 


By worker we mean those who are employed, or those who would be employed if 
they had sufficient skills to gain employment, i.e. those who want to work. 

Includes full-time and part-time employment. 


Intervention 


Skill 

development 

activities 


Learning programs referred to as deliberate general education or specific 
vocational activities undertaken to learn new skills or to further develop skills 
related to work. 

‘Deliberate’ in this context refers to the mature-aged worker knowingly participating 
in the skill development activity; for example, an activity which has a 
predetermined plan and format designed to develop employment-related skills and 
competencies. 

Deliberate skill development activities thus can include a structured (formal) 
learning activity, such as undertaking an accredited or non-accredited course, or 
an informal learning activity, such as self-directed learning, networking, coaching 
or mentoring. 


Outcome 1 


Improved 
attachment to 
the labour 
market 


Employee has improved attachment to the labour force. To include increased 
labour force participation rate, increased employment duration, job retention or 
improved job security (no longer at risk of losing job), change of work role or new 
career following new interest, change in attitude to work, to ‘early retirement’ or to 
‘retrenchment’ e.g. looking forward to continuing participation in paid work past 55 
years. 


Outcome 2 


Improved 

productivity 


Employer derives gain from increased activity from employee. 

Includes: increased wages, promotion, increased efficiency in processes and work 
output, reduction in accidents and injuries to workers through improved health and 
safety. 

Outcomes do not include participation in the voluntary sector. 



What we learnt 

The essence of the systematic review process is that it identifies only studies in which we can have 
confidence; that is, they meet both criteria of having findings which are of relevance and quality to 
answer the review question. In the appraisal phase, we considered the relevance of the findings and 
found (as anticipated) that there were many publications selected that did not really examine this 
particular review question, but rather made assumptions about outcomes of skill development 
activities for the mature-aged. 

Therefore, in future systematic reviews, the inclusion criteria must be applied more strictly to the 
full documents to ensure that only research studies which provide evidence to answer the review 
question are included in the in-depth review process. 

We recommend an additional selection stage for future reviews. The initial selection will be done 
by the searchers, using the abstracts, and will tend towards inclusion. The final selection process 
will be undertaken by the key reviewer and a second reviewer. Where necessary, the full document 
will be examined and assessed for relevance to the review question. Any studies which obviously do 
not meet the quality criteria for research should also be excluded at this stage. The aim would be to 
arrive at around 20 studies whose findings are most relevant to the review question and where the 
quality of those findings is high. Excluded studies may be referred to for useful contextual or 
background information in the final report, but will not be included in the in-depth review process. 

The steering and consultation group will be provided with lists of the excluded and included studies 
and asked to suggest further studies to be considered for inclusion. While it is useful to consult to 
ensure no key studies are missed, the final decision on inclusion must be made by the review team. 
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Conclusions 

Step 4: Select studies to be included 

A The inclusion criteria will be applied more strictly to the full documents by the key reviewer and 
second reviewer who will be involved in the final screening stage. This will include an initial 
appraisal of the relevance of findings to the review question and of the quality of the research. 

A Only research studies which provide evidence to answer the review question and which meet the 
quality criteria will be included in the in-depth review. There should be no more than 20 studies 
included for in-depth review (there may be a set of ‘reserve’ documents or lower rated studies 
kept for contextual information). 

A The reviewers who will be synthesising the evidence for the report will be familiar with all the 
included studies and aware of those excluded from the in-depth evaluation. 

A Sufficient time will be allowed so that references within the selected studies can be followed up, 
in order that these may be considered for inclusion in the review process. 

A All selected reviewers will attend a training workshop before commencing the in-depth appraisal 
and review of evidence from the selected studies. Detailed guidelines will be provided to the 
reviewers. 
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Appraising selected studies 



What we did 

The fifth step of the systematic review, as shown in figure 4 in appendix A, is the appraisal of the 
included studies. This required the application of the reviewers’ template as described in the earlier 
section; Developing a framework. Reviewers were selected and brought together for a training 
workshop, following which the draft template was modified and more detailed guidelines for 
appraisal of relevance and quality were developed. 

Recruitment and selection of reviewers 

Expressions of interest were called for external reviewers for the first NCVER systematic review. An 
advertisement was placed on the NCVER website with invitations also sent to researchers associated 
with the NCVER and other professional research organisations, such as the Australian Vocational 
Education and Training Research Association and the Australasian Evaluation Society consultants 
list. The reviewers were offered a set fee for each review (estimated to take on average one day per 
review) and had to be available to attend the training workshop and to have time for reviewing six 
documents within the next six weeks. Using the two selection criteria, research evaluation 
experience and expertise in the topic, the NCVER project team and steering group selected ten 
external reviewers for the first systematic review. 



Training of reviewers 

NCVER project personnel and external reviewers attended the one-day training workshop as well as 
one representative from ANTA, and one from the Commonwealth Department of Science, 
Education and Training. 

The session started with a general introduction to systematic reviews and an explanation of how 
NCVER was implementing the process. The reviewers were then presented with an overview of the 
searching process for the review: keywords and databases used; initial results. Using one of the 
studies included in the NCVER internal pilot, the reviewers were taken through the practical 
exercise of completing the reviewer’s template. The group discussed the application of the quality 
criteria and the need for further guidelines for this process. The revised reviewer’s evaluation 
template, the guidelines for appraisal, and a detailed task list were sent electronically to all reviewers 
after the workshop. 

At the end of the workshop, six studies from the mature-aged review List A studies were assigned to 
each reviewer. The reviewers were asked to complete their first assigned review and report back to 
the NCVER project coordinator within one week. On successful completion, and with feedback 
from questions which arose being sent to all reviewers, the reviewers proceeded with the other 
allocated studies. 

Allocation of studies 

Each study included in the systematic reviews was allocated to a pair of reviewers. Using the 
evaluation template, the two reviewers completed their review for each study independently and 
then advised the other when they had completed it. They then exchanged their independent 
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evaluations and commenced negotiations on an agreed combined review. Copies of all three reviews 
for each study were then sent to the NCVER project coordinator. 

The same two reviewers did not review more than one study together. That is, to arrive at an agreed 
combined review, each reviewer had to contact a different reviewer for each of their allocated 
studies. This method of allocating studies, where the independent reviewers were selected to avoid 
the same pair working together on more than one study, was chosen by NCVER to reinforce a 
replicable process, to reduce bias and to diversify the differing expertise of the reviewers. 



Coordination of review process and support for reviewers 

The NCVER project coordinator handled all questions from reviewers during the review process 
and responses were formulated and sent to all reviewers. After each reviewer had completed their 
first study and all questions were clarified, the reviewers continued the review process until all 
studies included in the systematic review were completed. 

In one or two instances, the reviewer had difficulty completing their allocated studies and so other 
trained reviewers were asked to review these. 

Outcome of appraisals 

Using a five-point scale (see table 4), each reviewer arrived at an overall weighting for the relevance 
of the particular focus of the study for addressing the question of the review. This included 
specifically considering the population, interventions and outcomes of the study compared with 
those defined in the systematic review framework in appendix A. 



Table 4: Summary of relevance appraisal 



In this research study: 






Rating 






Population i.e. mature-aged 


High 


Medium+ 


Medium 


Medium- 


Low 


Intervention i.e. skill development activities 


High 


Medium+ 


Medium 


Medium- 


Low 


Outcomes i.e. improved attachment to the 
labour market or productivity 


High 


Medium+ 


Medium 


Medium- 


Low 


Overall weight of evidence A (relevance) 


High 


Medium+ 


Medium 


Medium- 


Low 



Similarly, using a five-point scale as shown in table 5 and the guidelines for systematic review 
appraisal (appendix E) to answer the questions on validity, authenticity, reliability, sufficiency and 
currency of the research, reviewers arrived at an overall weighting for the quality of the research in 
relation to the evidence found to answer the systematic review question. 



Tables: Summary of quality appraisal 



In this research study: 






Rating 






Is the evidence valid? 


High 


Medium+ 


Medium 


Medium- 


Low 


Is the evidence reliable? 


High 


Medium+ 


Medium 


Medium- 


Low 


Is the evidence authentic? 


High 


Medium+ 


Medium 


Medium- 


Low 


Is the evidence sufficient? 


High 


Medium+ 


Medium 


Medium- 


Low 


Is the evidence current today? 


High 


Medium+ 


Medium 


Medium- 


Low 


Overall weight of evidence B (quality) 


High 


Medium+ 


Medium 


Medium- 


Low 



The project team then extracted the overall weights of evidence for relevance and quality from the 
combined reviews of all included studies and placed them in a matrix, as shown in table 6. 
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Reflection by reviewers of the review process 

Feedback on the review process was deliberately sought by NCVER from all reviewers, with an 
evaluation form sent to all reviewers to obtain their opinion on the process and changes for future 
systematic reviews. Most reviewers valued their involvement in the process and were keen to be 
involved again. Comments included how the process had influenced the reviewer in such a way as 
to change their approach to writing future primary research reports. The synthesiser for the mature- 
aged review, who was chosen from the team of trained reviewers, noted that it would have been 
helpful to have been involved in the process from the beginning — to have heard the arguments 
which led to the development of the review question. 



What we learnt 

It was very important to offer support and maintain communication with reviewers throughout the 
process. Reviewers worked independently and appreciated frequent messages of explanation and 
interpretation of what was a new process for most. A frequently asked question-and-answer file was 
compiled as a result of the first review and distributed to the reviewers. 

Comprehensive guidelines for the completion of the evaluation template were essential and will 
need adjustment in response to reviewers’ feedback. For example, a number of reviewers did not 
weight equally the five questions for appraising the quality of the study. This was not discussed at 
the 2004 training workshop. Thus, independently, some reviewers weighted one of the five aspects 
of quality as of most importance, for example, sufficiency or currency, and so their overall 
weighting followed the rating for that particular component. The project team was involved in 
moderating the overall weight of evidence rating in three cases in the first review (see table 6). This 
was required because the reviewers were unable to reconcile their weightings, or subsequently, the 
synthesiser found no evidence of relevance to the main review question. 

Lessons learned from the 2004 review will be valuable in training at the next workshop; the project 
team and experienced reviewers will know of the ‘sticking points’ and be able to give relevant advice. 

Selection and training of reviewers to undertake the appraisal of the selected studies is very 
important. Reviewers with particular technical expertise (for example, in quantitative research 
methods, statistical analysis, economic modelling or qualitative research methods) will be identified. 
NCVER will keep ongoing records of existing and potential reviewers. 

Reviewers will be given details of the searching and selecting processes and outcomes and asked to 
recommend studies for consideration or inclusion. 

Reviewer pairs will be assigned to included studies, although it is intended that around 20 studies 
will be selected for in-depth review in future. The collaborative and unbiased nature of evaluating 
research studies will be maintained. 

A full-day workshop will be held to train reviewers. A small study will be chosen as an example to 
work through on the day. Papers, including the study, will be circulated to reviewers beforehand. 
Reviewers will work in pairs and will have ample opportunity to discuss any concerns and make 
suggestions. Reviewers with experience may be asked to contribute to the running of the workshop. 
Studies for the actual review will be allocated at the workshop. 

The database containing the evaluation reports of each study did not eventuate in time to be used 
for the synthesis of the findings in the 2004 systematic review. However, it is believed that such a 
database would be useful in future reviews and further development of software to manage the 
systematic review documents is planned. 

Since undertaking the first review, we have investigated the software developed to manage 
comprehensive systematic reviews in health care by the Joanna Briggs Institute. This institute is an 
international research and development agency based in Adelaide, an affiliated research centre of 
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the University of Adelaide and the Royal Adelaide Hospital, and linked to an international 
collaboration of autonomous speciality, country and state-based collaborating centres — The Joanna 
Briggs Collaboration. 



Conclusions 

Step 5: Appraise the studies 

A Reviewers should be allocated studies, to some extent, according to their expertise in both the 
topic and the research and analytical techniques, as required (for example, quantitative research, 
including statistical analysis and economic modelling or qualitative research). 

A Each study included in the in-depth review will be allocated to two reviewers who, working 
independently, will enter into the electronic template their appraisal of the relevance and quality 
of the findings to the review question. The reviewers will then reach a consensus decision. 

A The project team may moderate ratings given by reviewers where consensus is not possible or 
some inconsistency is noted. 

A The reviewers will add to the database the details of study aims, methods, population, 
intervention, outcomes and findings and the best examples to illustrate the findings. 
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Synthesising the evidence 



The sixth step of the systematic review process is the synthesis of the findings from the included 
studies into a final report, thereby answering the review question. The associated report of the first 
systematic review (Thomson et al. 2005) was developed by the project team to describe the 
evidence available to answer the specific policy question. In doing this, issues for further research 
were raised. 

What we did 

Appointment of the synthesiser 

One of the external reviewers was selected on the basis of experience and availability as the 
synthesiser for the report on this first systematic review. As with other NCVER team members, 
before commencing this task, the synthesiser undertook a great deal of independent reading on 
systematic reviews to acquire background information. 

Identification of key studies and findings 

The appraisal results of the 33 combined reviews were placed in a matrix based on their relevance 
(weight of evidence A) and quality of their findings (weight of evidence B) to answer the review 
question. The appraisal results are summarised in table 6 and the referenced studies are listed in 
detail in the associated report (Thomson et al. 2005). 



Table 6: Appraisal matrix of 33 studies appraised 







Weight of evidence B (quality) 


High 


Medium plus 
(+) 


Medium 


Medium 
minus (-) 


Low 


Weight of 
evidence A 
(relevance) 


High 


91 




45 






Medium plus (+) 


(2), 90 


85 


(74) 






Medium 






(68) 


4, 49, 89 


18, 33* 


Medium minus {-) 


78, 81 


44, (57), 
(73*) 


(17), 83, 
(88*) 


6, 29, 36, 87 


38 


Low 


86 




10, 14 


66, 70, 77 


20, 43 



Notes: Study ID in bold contains evidence for review question and study ID in brackets contains information on barriers and facilitators. 

* Indicates those studies whose overall rating was moderated by the project team. 



For the synthesis of the evidence in answer to our review question, we used those studies with at 
least medium minus weighting of relevance to the review question and at least medium weighting 
based on the quality criteria. Seven studies were identified to include evidence for the review 
question (91, 90, 2, 85, 45, 74, 68 — marked in bold on the matrix). Three of these (2, 74, 68 — 
marked in brackets) specifically mentioned the existence of barriers and facilitators to skill 
development activities improving attachment to the labour market or productivity. The steering 
group had requested that such factors, if identified, should also be noted in the final report. In 



NCVER 



33 





addition, in looking at those studies appraised as medium minus for relevance, four more studies 
(57, 73, 17, 88 — marked in brackets) were identified as providing evidence for the existence of 
barriers and facilitators, although not providing evidence for the specific review question. These 1 1 
studies were used as the key studies in the synthesis of key findings for the associated report 
(Thomson et al. 2005). 

The evidence 

The project team developed a draft map for an NCVER-style report of the findings of the first 
systematic review. It included an introduction with background to the review question and the 
systematic review approach used. The findings chapter of the associated report outlines the evidence 
found in the research studies in whose findings we can have confidence for answering the specific 
review question. A further chapter provides the evidence found for the existence of barriers (and 
thus identifying possible facilitators) to the mature-aged undertaking skill development activities 
leading to improved attachment to the labour market or productivity. Finally, implications for 
further research, policy and practice are outlined in the report (Thomson et al. 2005). 



What we learnt 

It was at this stage of the review that we learnt that we should have screened more effectively (as 
outlined at Step 4), to eliminate studies which do not provide evidence to answer the review 
question (rated low in relevance), from the in-depth evaluation stage. Similarly, if reviewers 
undertaking the final selection stage note reasons why the study will definitely not meet the criteria 
for quality of findings to answer the review questions, these studies should also be eliminated before 
the in-depth review is undertaken. 

In the first review, we erred towards inclusion, because to have eliminated any possibly relevant 
studies may have resulted in too few findings to write a report. However, experience has shown us 
that a more rigorous screening process could safely be applied at an earlier stage to increase 
efficiency and allow more time for appraising fewer studies in detail. 

In the first review, it was found necessary for the synthesiser to refer to the full text of each key 
study and not rely solely on the findings placed in the template by the reviewers. This required 
additional time. 

We learnt also that there may be a need for the project team to moderate ratings given by reviewers. 
This was required for two studies where the reviewers were unable to reconcile their ratings (studies 
88, 33) and one where no evidence was found to answer the review question, so the overall 
relevance rating given by the pair of reviewers was inconsistent with other studies. 

One of the most important lessons we learnt is that the synthesiser should be involved in the 
systematic review process from the beginning in order to have a clear understanding of the 
arguments and selection criteria leading to the specific review question, and participate in the 
development of the framework and the reviewers’ template. Thus, this key reviewer may also be 
able to contribute to the management of the review process (for example, communicating with 
other reviewers). 

Although for the first review the synthesiser was selected after the reviewing process had begun, it 
was agreed that a better model (for reasons stated above) would be to appoint the synthesiser as part 
of the review process from the beginning. 
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Conclusions 

Step 6: Synthesise the evidence 

A A database will be used in future to enable electronic sorting and amalgamation of evidence 
from the studies to assist in the synthesis of findings and in checking the evidence trail for the 
final report. 

A The key reviewer and a second reviewer will synthesise the evidence found to answer the review 
question into categories, pooling material from the studies in whose findings we can have 
confidence. 

A Using other members of the project team to provide feedback, the key reviewer will compile a 
draft final report of the evidence to answer the review question, and implications for policy, 
practice and research. 
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Presentation of findings 

What we did 

The draft final report was sent to steering group members and external reviewers for comment 
before being published by NCVER. 

The final report was distributed to stakeholders and placed on the website. For purposes of 
transparency details of the process were also published in this report. 

NCVER will seek opportunities to disseminate the findings of the reviews and also to share our 
learning about the review process. For example, two members of the project team presented a paper 
to the national Australian Association for Research in Education (AARE) conference in November 
2004 to inform other educational researchers about the results and the process of the first systematic 
review in vocational education and training in Australia. 

Strengths of the review 

The NCVER final report of the findings of the mature-aged systematic review was based on a small 
number of key studies, but these were the result of an exhaustive search and in-depth review by a 
collaborative team. While the findings from this limited number of studies are important, other 
strengths have been identified in the review process. These include the development of 
infrastructure for future systematic reviews, as outlined in this report. 

In addition, another strength is that this well-documented, systematic and transparent approach 
enables the review to be updated by the same or different authors. Subject to available resources, 
additional research on this topic will be identified and will be made available as a supplement on 
the website. 

Another important strength of this systematic review is the capacity building, through the 
transparent and collaborative approach, to improve the quality of future VET research in Australia. 
This review has provided policy-makers, other stakeholders and researchers with the opportunity to 
look at the quality of our vocational education and training research. For consultant reviewers, 
there were many positive outcomes as demonstrated in their feedback on the review process. This 
included the need for some to re-assess their ideas about they way they write their research reports 
or articles for publication. It also caused them to re-assess the nature, purposes and scope of 
literature reviews, and the role of referees and editors. 

What we learnt 

We learnt from the first systematic review that it may be that only a small number of studies will be 
identified as key studies; that is, studies in which we can be confident of the findings, because they 
rate highly for relevance and for quality. The first review found seven key studies to provide 
evidence to answer the review question and used another four studies with additional supporting 
evidence for barriers and facilitators. The synthesis report was based on these 1 1 key studies, 
although it may refer to other studies for context or other supporting evidence. 
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The systematic review also provides an opportunity to identify gaps in current research and 
implications for further research, policy and practice. Exhaustive identification and evaluation of 
existing research enabled us to say that, in the case of the mature-aged review, further research is 
required into the question, especially about the nature of the skill development activities and their 
related outcomes. 

It is important to maintain the involvement of the reviewers who have contributed key work to the 
outcome of the systematic review. The steering group, reviewers and consultation group were kept 
involved throughout the systematic review. 



Conclusions 

Step 7: Present findings to stakeholders 

A The draft final report will be distributed to the steering group members and reviewers for 
comments before finalising for publication. 



Step 8: Disseminate the findings 

A The final report will be published on the NCVER website. 

A Presentation of the findings of systematic reviews will be made to stakeholders through research 
forums, conferences and other channels as appropriate. 
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Appendix A 
Framework for first 
systematic review 



Final framework for the systematic review of research: Skill 
development activities that keep the mature-aged in paid work 

The framework for the NCVER/ANTA systematic review of research includes: 

A the policy question(s) to be addressed by the review 
A the definitions of the terms in the question(s) 

A the criteria to be used to select research for inclusion in the review 
A the coding to be used for recording the findings from the included studies 
A the criteria for appraising the quality of the included studies 

All components of the framework must be considered in conjunction with each other when carrying out 
the review. 

A list of the steering group members and an outline of the review process are provided at the end of 
this framework document for reference. 

The policy question 

The question developed by the steering group to be addressed in this review is: 

What evidence is there that skill development activities for the mature-aged lead to: 

A improved attachment to the labour marked 
A improved productivity^ 

The steering group agreed that this question needs some unpacking. Implied in the review question, 
and to be reported in the research findings and final report, are the issues of which skill 
development activities work, when and for whom. Also to be reported in the findings are the factors 
(such as attitude) that have an important bearing on these key outcomes, either as barriers or 
facilitators, and the implications for policy, practice and research from these findings (see coding of 
findings section). 



The definitions of the terms in the questions 

The terms in the question as defined by the steering group are as follows. 

Mature-aged 

The starting brief contained the term ‘mature-aged worker’, but after discussion by the steering 
group the population was defined more broadly as ‘mature-aged’. 

Mature-aged refers to all people 45 years and over. It includes all those who desire work, or who 
might desire work if their skills were more suitable or if there were more opportunity. 
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Skill development activities 

Skill development activities are defined as ‘deliberate’ general education or specific vocational 
activities undertaken to learn new skills or further develop skills related to work. ‘Deliberate’ in this 
context refers to the person knowingly participating in the skill development activity, for example, 
an activity which has a predetermined plan and format designed to develop employment-related 
skills and competencies. 

Skill development activities thus can include a structured (formal) learning activity such as 
undertaking an accredited or non-accredited course, or an informal learning activity such as self- 
directed learning, networking, coaching or mentoring, as long as it is predetermined or knowingly 
participated in. 

Skill development activities may be initiated by self, employer or other (such as Job Network 
member). 

Improved attachment to the labour market 

Improved attachment to the labour market includes: 

A increased labour force participation rate 
A increased employment duration 

A job retention or improved job security (no longer at risk of losing job) 

A change of work role or new career following new interest (including self-employment) (what’s 
driving longer engagement with the labour market?) 

A change in attitude to work, ‘early retirement’ or ‘retrenchment’, for example, looking forward to 
continuing participation in paid work past 55 years. 

Improved productivity 

Improved productivity includes: 

A increased wages 
A promotion 

A increased efficiency in processes and work output 

A reduction in accidents and injuries to workers through improved health and safety. 

Outcomes do not include participation in the voluntary sector. 

Research selection criteria 

The search strategy will involve hand and electronic searching (databases and websites), to select 
studies that: 

A are in English (from Australia and overseas) 

A are recent (from approximately the last 10 years) 

A match keywords deemed relevant to the review question. 

Note, however, that research dealing with a population under 45 years will also be included where 
the research findings are relevant to improving attachment to the labour force for the mature-aged. 
Similarly, while the research is likely to cover those up to 65 years, relevant research focusing on 
over-65-year-olds will also be included. 

All found studies will be recorded with reasons for any exclusion noted, such as because they do 
not answer the set question, or a study has been replaced by a primary research study on which it 
was based. 
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All included studies will undergo a further coding process as outlined below. 



Coding of the findings from the included studies 

As well as defining the question for the review, the steering group and others who provided 
feedback on drafts were concerned to ensure maximum value from the review by specifying how the 
findings of included research were to be recorded. 

Findings from the included studies will be recorded and the studies coded by the reviewers within 
categories as suggested in table 7. Note that more than one code may apply, and some categories 
will be open-ended and more terms may be added as needed to code the studies systematically. 



Table 7: Coding of the included studies 



Item 


Details 


Bibliographic details, including: 
title 

author(s) 
publication details 
series 
uri 


Title, author(s), publication date and other 
relevant bibliographic details 


Aims of the research 


Brief description of study aims, including 
research questions and any hypotheses 


Study methodology (proposed and actual) 
<• textual description 


Brief textual description of the proposed 
methodologies for the study, including 
details of the sample size, population and 
coverage 




Any differences between the actual and 
proposed methods used will be noted 


Geographic location 
local (Australian) 

<• state (Australian) 

<• national (Australian) 

international (one country, not Australia) 
^ international (more than one country) 


Location the study covers 


Time period 

date range, by year 


Time period the study covers 


Mature-aged population 
45 to 65 
over 65 

under 45 but relevant to over 45 


Size and nature of the mature-aged 
population in the study. This will include 
the population from which the sample was 
drawn as well as the sample itself 


Other terms may be needed, arising from the studies included. 




Skill development activities (the five categories here will be further 
subdivided, as indicated by the examples): 

provider (for example, school, TAPE, higher education, adult and 
community education, private training provider, enterprise training) 
delivery (for example, on-the-job training, off-the-job training, 
apprenticeship, traineeship, distance education, e-learning, in 
classroom, informal) 

^ field of education (Australian Standard Classification of Education 
categories) 

level of qualification (AQF and non-AQF, for example, non-award, 
subject only, cert l-IV, diploma, bachelor degree or higher) 
duration (for example, short-term, long-term) 


Description of skill development activities, 
to include but not be limited to: education 
sector, formal or informal, duration 


Other terms may be needed, arising from the studies included. 




Outcomes (the 2 categories here will be further subdivided, as 
indicated by the examples): 

any changes in attachment to the labour market (for example, in 
labour force participation, employment duration, job retention, job 
security, change of work role, change of career, change in work 
attitude, re-entry to labour market, move to or from self- 
employment) 

any changes in productivity levels (for example, wages, promotion, 
demotion, efficiency, output, health and safety) 


Description of the outcomes (in relation to 
attachment to the labour market and/or 
productivity) of the skill development 
activity 


Other terms may be needed, arising from the studies included. 
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Item 



Details 



Barriers or facilitators: 

<)■ attitudes as barrier/facilitator 
<)■ other social/cultural factors as barrier/facilitator 
<> financial circumstances as barrier/facilitator 
<> health as barrier/facilitator 
<> disadvantage (equity group) as barrier/facilitator 
<> access to services (educational or other) as barrier/facilitator 
<> alternative opportunities as barrier/facilitator 
<> prior educational background as barrier/facilitator 
<)■ area of residence as barrier/facilitator 

Other terms may be needed, arising from the studies included 

Findings Findings as described by the author will be 

recorded 

Relevant additional comments, for 
example, about the context of the study, 
supplied by a reviewer or other external 
expert. 

Appraisal of the included studies 

To evaluate the studies, there will be two weights of evidence allocated to each study rated on a 
five-point Likert scale. 

Weight of evidence A: Reviewers will rate the relevance of the particular focus of the study for 
addressing the review question (considering the population, intervention and outcomes, as 
described in this framework, compared with those described in the study). 



Table 8: Assessment criteria for weight of evidence A (relevance) 



Rating 


Population i.e. mature-aged 


High 


Medium+ 


Medium 


Medium- 


Low 


Intervention i.e. skill development activities 


High 


Medium+ 


Medium 


Medium- 


Low 


Outcomes i.e. improved attachment to the 
labour market or productivity 


High 


Medium+ 


Medium 


Medium- 


Low 


Overall weight of evidence A (relevance) 


High 


Medium+ 


Medium 


Medium- 


Low 


Weight of evidence B: Reviewers will rate the quality of the study in terms of the trust that can be put 
in its findings against the questions posed (considering the rules of evidence criteria: validity, 
reliability, authenticity, sufficiency, and currency). 


Table 9: Assessment criteria for weight of evidence B (quality) 








In this research study: 






Rating 






Is the evidence valid? 


High 


Medium+ 


Medium 


Medium- 


Low 


Is the evidence reliable? 


High 


Medium+ 


Medium 


Medium- 


Low 


Is the evidence authentic? 


High 


Medium+ 


Medium 


Medium- 


Low 


Is the evidence sufficient? 


High 


Medium+ 


Medium 


Medium- 


Low 


Is the evidence current today? 


High 


Medium+ 


Medium 


Medium- 


Low 


Overall weight of evidence B (quality) 


High 


Medium+ 


Medium 


Medium- 


Low 



Note: Both weighting A (relevance/focus compared with review question) and weighting B (quality/reliability of the findings of the 

study) will be taken into account in preparing the final synthesis report. 



Expert comment 



Description of barriers/facilitators (to 
include how other factors might have 
impacted on the effectiveness of the skill 
development activities such as attitudes, 
financial circumstances, health, availability 
of alternative careers, or socio-economic 
context) 
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Reporting 

Full reporting will detail: 

A number of studies found in the initial search 
A number of studies excluded, and reasons for exclusion 
A full details of results of appraisals of included studies. 

The final synthesis phase will answer the review question by detailing the evidence (and weight) for 
skill development activities for the mature-aged that lead to improved attachment to the labour 
market and/or improved productivity. It will state explicitly what skill development activities work, 
under what circumstances, and for which groups of the mature-aged, noting barriers and 
facilitators, and detailing the implications for policy, practice and research. 



Background 

The National Centre for Vocational Education Research received funding from the Australian 
National Training Authority to undertake, for the first time, a systematic review of research for 
Australian VET policy-makers on a question of policy salience. Systematic reviews in the field of 
education have been undertaken overseas since the late 1990s but this method has not previously 
been used in Australian education research. In the course of undertaking the review NCVER will 
also be developing a framework and capabilities for similar reviews. 

The outcomes of the first NCVER systematic review of research project to be achieved by the end 
of 2004 are: 

A a report of the systematic review of research into a key policy question that ANTA has specified 
is to be related to mature-aged workers 

A a replicable framework and infrastructure within which further systematic reviews of research 
can be conducted 

A a report on the process of this systematic review of research. 

In undertaking this review, NCVER is seeking a highly interactive relationship with 
Commonwealth, state and territory policy-makers, and a network of VET researchers throughout 
the systematic review process. Consultation will ensure that this first Australian VET systematic 
review of research has current relevance to policy- and decision-makers. 

What does a systematic review of research involve? 

A systematic review of research is a secondary research activity that locates all relevant existing 
material (published and unpublished) on a focused policy question. It evaluates this material for its 
information content, approach and robustness, concluding with a balanced and relevant synthesis 
of the findings. 

A systematic review of research follows a structured framework, and is transparent in its approach, 
making clear the criteria and reasons why a study has or has not been included, and the basis for the 
judgement of its quality. Thus it provides an empirically based foundation for decision-making. 

The United Kingdom-based Evidence for Policy and Practice Information and Co-ordinating 
Centre was established in 1993 to address the need for a systematic approach to the organisation 
and review of evidence-based work on social interventions. The centre-developed model of 
systematic reviewing has been followed and adapted by the Learning and Skills Development 
Agency in the United Kingdom. In this project, NCVER will adapt the centre model and other 
approaches to suit our purposes. The eight steps for this systematic review are outlined in the 
following chart. 



NCVER 



43 




Iterative process 



Figure 4: Summary of the steps of a systematic review 
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► 



► 
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Report 
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Appendix B 
Resources used for search 



The systematic searching for relevant literature was undertaken during May and June 2004. 

Electronic online databases searched 

Social SciSearch (via database vendor - Dialog) 

Education Abstracts (via database vendor - Dialog) 

ERIC (via database vendor - Dialog) 

EconLit (via database vendor - Dialog) 

Australian Education Index (AEI) cdrom + online via Informit Online 
British Education Index (BEI) cdrom 

Web based databases searched 

VOCED (Vocational Education and Training Research Database) <http://www.voced.edu.au> 
AgeLine (American Association for Retired Persons) <http://research.aarp.org/ageIine/home.htmI> 
EconPapers Online <http://econpapers.hhs.se/> 

Educational Research Abstracts (ERA) online <http://www.tandf.co.uk/era/> 

Web search engines used 

Google <http://www.googIe.com.au> 

Vivisimo <http://vivisimo.com/> 

Web sites viewed 

Ageinfo Information service (UK Centre for Policy on Ageing) <http://www.cpa.org.uk/ageinfo/ 
ageinfo.htmI> 

Ageing Policy Home (Uni of Melbourne, National Clearinghouse Project — Ageing policy) 
<http://www.pubIic-poIicy.unimeIb.edu.au/research/CIearinghouse/Ageing/Ageing.htmI> 

GeroNet - Center for Policy Research in Aging <http://www.geronet.med.ucIa.edu/centers/cpra/#res> 
Barbara McIntosh Publication History — School of Business Administration, University of Vermont 
<http://www.bsad.uvm. edu/Research/FacPubs/detaiIs?author=62> 

Centre for Reviews and Dissemination - University of York <http://www.york.ac.uk/inst/ 
crd/index.htm> 

United Nations: Policies and programmes on ageing <http://www.seniorweb.nl/un/start.asp> 

English Longitudinal Study of Ageing <http://wwwI.ifs.org.uk/elsa/index.htm> 

The Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) 
<http://eppi.ioe.ac.uk/EPPIWeb/home.aspx> 

Eurofound: Active Strategies for an ageing population <http://www.eurofound.eu.int/Iiving/ 
ageing.htm> 

Evidence Network <http://www.eurofound.eu.int/Iiving/ageing.htm> 



Journals - content pages online 

Ageing & Society <http://titles. Cambridge. org/journaIs/journaI_cataIogue.asp?historyIinks= 
ALPHA&mnemonic=ASO> 
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Appendix C 
Key search terms 



Note: mature age(d) was not a term used in the VOCED, ERIC, AEI, BEI or AgeLine Thesauri. 

Population 

older worker / older workers / older adults / older students / older learner / older learners / old age / 
middle aged adults / old old adults / young old adults / ageing / aging / aging (individuals) / ageing 
population / aging population / aged / adult student / adult learner / adult learners / mature worker 
/ mature workers / mature age worker / mature age workers / mature-age worker / mature-age 
workers / mature aged worker / mature-aged workers / mature-aged worker / mature- aged workers / 
mature learner / mature learners / mature age learner / mature age learners / mature-age learner / 
mature-age learners / mature-aged learner / mature- aged learners / mature-aged learner / mature- 
aged learners / baby boomers / elderly / 50+ / population ageing / population aging 

Intervention 

training / education / learning / learning activity / deliberate general education / vocational training 
/ vocational education / continuing education / continuing vocational education / continuing 
vocational education / adult learning / adult education / adult programs / adult vocational 
education / lifelong learning / retraining / refresher training / job skills / job development / job 
training / skill development / skill upgrading / skills / accredited / unaccredited / self-directed 
learning / formal learning / networking / coaching / mentoring / reentry student / human capital / 
training investment / intervention / interventions 

Outcomes 

participation / participation rates / productivity / alternative work patterns / postretirement work / 
employment / engagement / employment opportunity / part-time employment / full-time 
employment / job retention / job security / job satisfaction / career change / new career / entry to 
job market / entry to labour market / entry to labor market / longer engagement with labour 
market / longer engagement with labor market / reduced retrenchment / promotion / increased 
hours or work / regular hours or work / labour utilisation / labor utilisation / labour market / labor 
market / labour force development / labor force development / labour force participation / labor 
force participation / labour productivity / labor productivity / educational benefits / educational 
attainment / no early retirement / reduced accidents / improved health and safety / return on 
investment / tenure / return to work. 
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Appendix D 
Evaluation template 



Systematic review of research: Skiii deveiopment activities 
that keep the mature-aged in paicf work 

REViEWER Evaiuation Tempiate 



To ensure consistency and maximum value from the review, the following template has 
been developed for completion by each reviewer. It is based on the Framework for the 
systematic review of research, Skill development activities that keep the mature-aged 
in paid work. For your response please circle or bold the relevant answer(s). 

Important notes: 

1 . For all fields, if you are supplying information that you have inferred from the 
review, or ascertained from other sources, please make this clear by putting it in 
square brackets and writing [inferred by reviewer] at the end of the comment or 
code. It is necessary to distinguish between what is explicit in the report and what is 
implicit, or sourced from elsewhere. You may need to undertake further 
investigation (eg a phone call to an author) to answer a question. If that is the case 
it must be recorded. 

2. Wherever necessary, put a page reference to indicate where in the report you 
found the text on which your decision was based. This will help in the discussion 
with your fellow reviewer and in the final synthesis. 

3. In many fields, you are asked to indicate one or more codes that apply to all or part 
of the study. You are also able to use a text field to write comments about how the 
codes apply; for example if the study uses several types of methodology in different 
geographic areas, you can outline exactly which codes apply and how they are 
related. 



1. ABOUT THE STUDY YOU ARE REVIEWING 



Q1 Information needed for this section will be supplied with the studies. 



1.1 ID: 

1 .2 Author(s): 

1.3 Title: 
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1.4 Publication details (place, 
publisher and date): 

1.5 Source (detail below): Web Hand search Database 

1.6 Details: 

2. ABOUT THE REVIEWER 

Q2 Please complete your details below: 

2.1 Name(s): 

2.2 Date of review: 

3. ABOUT THE RESEARCH, IT’S SCOPE AND ITS FINDINGS 

Q3.1 Briefly describe the aims and scope of the study, including research questions 
and any hypotheses set up and tested by the study. Use your own words to 
paraphrase only if necessary (for example if the study describes its aims 
differently in several places). 
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Q3.2 State all the key findings and how they relate to the objectives of the study. 
Please use a separate paragraph for each finding. Use your own words to 
paraphrase only if necessary (for example if the study describes its findings 
differently in several places). 




Q3.3a Please describe all the proposed methodologies for this study. Include details of 
the sample size, population and coverage, (e.g. 30 in-depth interviews in NSW, 
2 focus groups in ACT, and a national telephone survey of 400 mature-aged) 




Q3.3b Please describe any differences between the actual and proposed methods 
used for this study. 
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Q3.4 What geographic location(s) does the actual study cover? Select all that apply 
and make comments in your own words below if you need to explain different 
geographic locations applying to different parts of the study. Note: local can 
apply to a location within a particular state, organisation or industry. Please list 
the region(s)/state(s)/countries the study covers, and make explanatory notes if 
necessary 



3.4a 


Australia (national) 


3.4b 


Australia (state) 


3.4c 


Australia (local) 


3.4d 


International (one country only, not Australia) 


3.4e 


International (more than one country, 
may include Australia) 


3.4f 


Other (detail below) 




Q3.5 What time-period does the study cover? Make explanatory notes if necessary 



3.5a Month(s) if relevant: 

3.5b Year (inc range if relevant): 




Q3.6 What age of mature-aged population does the study cover? (Use more than one 
if needed, indicate if only part of range covered); make explanatory notes if 
necessary. Please record here both the actual sample used and the expected 
sample, if different, and provide an explanatory note in your comments. If the 
population from which the sample was drawn is different from the actual 
sample, please also record that in your comments. 



3.6a 


Over 45 


3.6b 


Over 50 


3.6c 


45-65 


3.6d 


Over 65 


3.6e 


Under 45 but relevant to over 45 


3.6f 


Other (detail below) 




50 



An aid to systematic reviews of research in vocational education and training in Australia 








Q3.7 If there are any other particular characteristics of the included population that 
you have not yet recorded (e.g. gender, employment status), please detail 
below. List the characteristics and make explanatory notes if necessary. Please 
record here both the actual sample used and the expected sample, if different, 
and provide an explanatory note in your comments. If the population from which 
the sample was drawn is different from the actual sample, please also record 
that in your comments. 



4. ABOUT THE SKILL DEVELOPMENT ACTIVITIES 



Q4.1 Who provided the skill development activities? Make explanatory notes as 
necessary. List any other provider(s) and give details. 



4.1a 


School 


4.1b 


TAPE 


4.1c 


Adult & community education (ACE) 


4. Id 


Private provider 


4.1e 


Enterprise training 


4. If 


other (detaii beiow) 
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Q4.2 How was the skill development activity delivered? Select all that apply. Make 

explanatory notes as necessary. List any other delivery methods/types and give 
details. 



4.2a 


On-the-job training (structured) 


4.2b 


On-the-job training (unstructured) 


4.2c 


Off-the-job training (externai) 


4.2d 


Off-the-job training (in-house) 


4.2e 


Apprenticeship 


4.2f 


Traineeship 


4.2g 


Distance education 


4.2h 


On-iine (e -iearning) 


4.2i 


In ciassroom 


4.2j 


In informal community setting 


4.2k 


other (detaii beiow) 







Q4.3 What was the nature of the setting for the skill development activity? Make 
explanatory notes as necessary. List any other setting(s) and give details. 



4.3a 


Formal (structured) 


4.3b 


Informal (includes self-directed, networking, 
mentoring, coaching) 


4.3c 


Formal & informal 


4.3d 


other (detail below) 



Q4.4 What were the major field(s) of education for the skill development activity? Use 
ASCED (2 digit level) as set out below. Information about what is included in 
each of these is included on a separate sheet and is also available at 
http://www.abs.qov.au/Ausstats/abs%40.nsf/66f306f503e529a5ca25697e00176 
61f/41ca34c3a2af0c2cca256aaf001fca77!OpenDocument 

Select all that apply (if any). Make explanatory notes as necessary. 



4.4a 


Natural & physical sciences 


4.4b 


Information technology 


4.4c 


Engineering & related technology 


4.4d 


Architecture & building 


4.4e 


Agriculture, env. & related studies 


4.4f 


Flealth 


4.4g 


Education 


4.4h 


Management & commerce 


4.4i 


Society & culture 


4.4j 


Creative arts 
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4.4k 


Food, hospitality & personal services 


4.41 


Literacy & numeracy 


4.4m 


Learning skills 


4.4n 


Social skills 


4.4o 


Employment skills 


4.4p 


Other general education 


4.4q 


Other mixed field programs 


4.4r 


Subject only (no field of education) 



Q4.5 Indicate, if relevant, the level of qualification for the skill development activity(s). 
Select all that apply (if any). Make explanatory notes as necessary. 



4.5a 


Non-award course 


4.5b 


AQF Subject only/statement of attainment 


4.5c 


Secondary education cert. 


4.5d 


Certificate 1 


4.5e 


Certificate II 


4.5f 


Certificate III 


4.5g 


Certificate IV 


4.5h 


Diploma 


4.5i 


Advanced Diploma 


4.5j 


Graduate Certificate 


4.5k 


Graduate Diploma 


4.51 


Bachelor Degree or higher 


4.5m 


Other - specify below: 







Q4.6 Indicate, if relevant, the duration of the skill development activity(s). Make 
explanatory notes as necessary. 
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Q4.7 If there are any other features of the skill development activities that you have 
not yet recorded, please detail below, with any necessary explanatory 
comments. 




5. ABOUT THE OUTCOMES OF THE SKILL DEVELOPMENT ACTIVITIES 



Q5.1 Does the study identify and/or demonstrate change(s) in attachment to the 

labour market of the relevant mature-aged population as an outcome of the skill 
development activity/activities? 

1 Yes □ 2 No □ 3 Not applicable □ 

If yes, please indicate all that describe this change in attachment to the labour 
market. Make explanatory notes as necessary. List any other relevant outcomes 
and give details. Please note outcomes here do not include participation in the 
volunteer sector: 



5.1a 


Change in labour market participation 


5.1b 


Change in employment duration 


5.1c 


Job retention 


5.1d 


Change in job security 


5.1e 


Change of work role 


5.1f 


Job promotion or demotion 


5.1g 


Change of career 


5.1b 


Change in work attitude 


5.11 


Re-entry to labour market 


5.1j 


Move to or from self-employment 


5.1k 


other (specify below) 








Q5.2 Does the study identify and/or demonstrate improved productivity of the relevant 
mature-aged workers as an outcome of the skill development activity/activities? 



1 Yes □ 2 No □ 3 Not applicable □ 
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If yes, please indicate all that describes this improved productivity within the 
labour market. Make explanatory notes as necessary. List any other relevant 
outcomes and give details. 



5.2a 


Increased wages 


5.2b 


Promotion 


5.2c 


Increased efficiency 


5.2d 


Increased output 


5.2e 


Improved health & safety 


5.2f 


Increased morale 


5.2g 


Other (specify below) 








Q5.3 If the study identifies and/or demonstrates outcomes other than improved 

attachment to the labour market or improved productivity, as an outcome of the 
skill development activities, please describe below. 

Please include negative as well as positive outcomes. 




Q5.4 If there are any other details regarding the outcomes of the skill development 
activity(s), as described in the study, that you have not yet recorded, please 
detail below. Make explanatory notes as necessary. 
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6. ABOUT BARRIERS OR FACILITATORS TO SKILL DEVELOPMENT ACTIVITIES 
AND/OR THEIR OUTCOMES 



Q6.1 Does the study identify and/or demonstrate barriers to skill development 
activities for the mature-aged leading to improved attachment to the labour 
market and/or productivity? 

1 Yes □ 2 No □ 3 Not applicable □ 

If SO, please indicate what these barriers are (circle all that apply). Make explanatory 
notes as necessary. List any other relevant barriers and give details. 



6.1a 


Peer attitude 


6.1b 


Health 


6.1c 


Personal attitude 


6.1d 


Financial circumstances 


6.1e 


Employer attitude 


6.1f 


Availability of alternative careers 


6.1g 


Access to services (educational or other - 
specify) 


6.1h 


Disadvantage (equity group) 


6.11 


Prior educational background 


6.1j 


Geographic location 


6.1k 


Physical ability 


6.11 


Employment opportunities 


6.1m 

6.1o 


Socio-economic factors (specify below) 
others (specify below) 


6.1n 


other social/cultural factors 



Q6.2 Does the study identify and/or demonstrate facilitators to skill development 
activities for the mature-aged leading to improved attachment to the labour 
market and/or productivity (circle one only)? 

1 Yes □ 2 No □ 3 Not applicable □ 
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If so, please indicate what these facilitators are (circle all that apply). Make 
explanatory notes as necessary. List any other relevant facilitators and give 
details. 



6.2a 


Peer attitude 


6.2b 


Health 


6.2c 


Personal attitude 


6.2d 


Financial circumstances 


6.2e 


Employer attitude 


6.2f 


Availability of alternative careers 


6.2g 


Access to services 


6.2h 


Disadvantage (equity group) 


6.21 


Existing skills 


6.2j 


Geographic location 


6.2k 


Physical ability 


6.21 


Employment opportunities 


6.2m 


Socio-economic factors 


6.2n 


Prior educational background 


6.2o 


Social/cultural factors 


6.2p 


Others (specify below) 




Q6.3 If there are any other details regarding barriers or facilitators, as described in 
the study, that you have not yet recorded, please detail below. Make 
explanatory notes as necessary. 




7. ANY FURTHER COMMENTS ABOUT THE STUDY 



Q7 Please feel free to add expert comment on any additional aspect that you 

believe is relevant to this systematic review, for example about the context of 
the study or the funding source. 
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8. ANY OTHER RELEVANT RESEARCH 



Q8 Finally, please put details here of any highly relevant research referred to in the 
study you are reviewing, that you believe should be followed up for the 
systematic review. 



9. EVALUATION SECTION 



In this section we are interested in your evaluation of this study. Please refer to the 
Reviewers’ Guidelines for Quality Appraisal for explanatory detail. 



9.1 WEIGHT OF EVIDENCE A (relevance) 



Q9.1 How would you rate the relevance of the particular focus of the study for 

addressing the question of this review? Consider the population, intervention 
and outcomes as described in the review framework compared with those 
covered in this study. Rate each component as well as giving an overall rating. 



In this research study: 






Rating 






Population i.e. mature-aged 


High 


Medium+ 


Medium 


Medium- 


Low 


Intervention i.e. skill development activities 


High 


Medium+ 


Medium 


Medium- 


Low 


Outcomes i.e. improved attachment to the 
labour market or productivity 


High 


Medium+ 


Medium 


Medium- 


Low 


Overall Weight of Evidence A (relevance) 


High 


Medium+ 


Medium 


Medium- 


Low 


Please add any comment below: 
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9.2 WEIGHT OF EVIDENCE B (quality) 



Q9.2 How would you rate the quality of this study in terms of the trust that can be 
put into its findings against the questions posed? Please refer to the Reviewers’ 
Guidelines for Quality Appraisal. 



In this research study: 






Rating 






Is the evidence valid? 


High 


Medium+ 


Medium 


Medium- 


Low 


Is the evidence reliable? 


High 


Medium+ 


Medium 


Medium- 


Low 


Is the evidence authentic? 


High 


Medium+ 


Medium 


Medium- 


Low 


Is the evidence sufficient? 


High 


Medium+ 


Medium 


Medium- 


Low 


Is the evidence current today? 


High 


Medium+ 


Medium 


Medium- 


Low 


Overall Weight of Evidence B (quality) 


High 


Medium+ 


Medium 


Medium- 


Low 



Please add any comment below: 



9.3 WEIGHT OF EVIDENCE SUMMARY 



Q9.3 Please provide final weights of evidence A and B below: 



Weight of evidence A 



Weight of evidence B 



Thank you for completing this evaluation form. 
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Appendix E 
Guidelines for systemadc 

review appraisal 



Item: 
Criterion A: 

Guidelines: 


Weight of evidence A 

How would you rate the relevance of the substance of this study to the 
review question? 

Consider the following in answering the question above: 

1 . Actual population sample covered compared with that specified in the review framework 

2. Actual intervention(s) covered compared with that specified in the review framework 

3. Actual outcomes focused on compared with those specified in the review framework 


Item: 


Weight of evidence B 


Criterion B: 


How would you rate the quality of the study in terms of trust that can he 
put into its findings for the research question posed? 


Guidelines: 


Consider all quality criteria as outlined below i.e. validity, reliability, authenticity, sufficiency and 
currency 


Criterion 1; 


Is the evidence in this study valid? 


Guidelines: 


Consider the following in answering the question above: 

1 . research aims and variables or concepts measured 

2. design and whether methods measure what was intended to be measured (i.e. validity) 

3. efforts made to address the validity of data collection tools/methods e.g. pilot testing tools 

4. efforts made at data analysis stage to address validity e.g. limiting analyses where numbers 
are insufficient 

5. efforts made in the analysis to control for bias from confounding variables 

6. whether links between data, interpretation and conclusions are valid 

7. justification of conclusion drawn 


Criterion 2; 


Is the evidence in this study reliable? 


Guidelines: 


Consider the following in answering the question above: 

1 . efforts to determine that data collection methods and tools will yield same result each time 
(i.e. are reliable) 

2. efforts to ensure data analysis can be repeated and yields same result each time 

3. any assumptions/theoretical perspectives that shape the form or the output of the research 

4. any alternative explanations for stated findings 


Criterion 3: 


Is the evidence in this study authentic? 


Guidelines: 


Consider the following in answering the question above: 

1 . whose voice it is in the report 

2. for whom and for what purpose the knowledge was being sought 

3. sources of evidence: direct or indirect 
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Criterion 4; 


Is the evidence in this study sufficient? 


Guidelines: 


Consider the following in answering the question above: 




1. 


sample sizes etc. 




2. 


the authors’ conclusions 




3. 


whether there are any other possible explanations for the findings 




4. 


that the evidence presented is enough to support the findings and conclusions 


Criterion 5: 


Is the evidence in this study current today? 


Guidelines: 


Consider the following in answering the question above: 




1. 


when the study was done and any contextual issues that are no longer relevant or that now 
are relevant that were not at the time of the study 



The weight of evidence A & B will then be scored in a 5 x 5 matrix; 







Weight of evidence B (quality) 


High 


Medium plus 
(+) 


Medium 


Medium 
minus (-) 


Low 


Weight of 
evidence A 
(relevance) 


High 












Medium plus (+) 












Medium 












Medium minus {-) 












Low 
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A systematic review of research is a decision-making tool for policy 
and practice. It is a piece of research in its own right, using explicit 
and rigorous methods that follow a standard set of stages; to 
identify; critically appraise; and synthesise relevant research (both 
published and unpublished) to answer a specific question. 

This report documents the establishment of NCVER’s own eight- 
step model for future systematic reviews of research. 
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